Highlights §
- It seems best to specify a learning rate of
1e-4 to 1e-3, which is higher than normal DreamBooth or fine tuning. (View Highlight)
- —network_dim` * Specify the LoRA RANK (e.g.
--networkdim=4). If omitted, it will be 4. The greater the number, the more expressive power there will be, but the memory and time required for learning will also increase. It also doesn’t seem to be a good idea to increase the number blindly. (View Highlight)
--network_alpha * Specify the alpha value to prevent underflow and ensure stable learning. Default is 1. Specifying the same value as network_dim will result in the same behavior as in previous versions. (View Highlight)-
- Specify the number of processes for reading data. Having a large number of processes allows for faster data loading and more efficient use of the GPU, but it also consumes main memory. The default is “the smaller of
8 or Number of CPU concurrently executing threads - 1”, so if you don’t have enough main memory or the GPU usage rate is about 90% or more, use 2 while looking at those numbers. Or lower it to about 1. (View Highlight)
-
--network_weights * Load the learned LoRA weights before learning and additionally learn from them. (View Highlight)
--network_train_unet_only * Only LoRA modules related to U-Net are enabled. It may be a good idea to specify this using fine tuning learning. (View Highlight)- DyLoRA was proposed in this paper. DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation Official implementation [here](https://github.com/huawei -noah/KD-NLP/tree/main/DyLoRA). According to the paper, a higher rank for LoRA is not necessarily better, and it seems necessary to find a rank that is more appropriate for the target model, dataset, task, etc. Using DyLoRA, you can simultaneously learn LoRA at various ranks below the specified dim(rank). This saves you the trouble of learning and searching for the optimal rank. The implementation in this repository is based on the official implementation with original extensions (therefore, there may be some defects). (View Highlight)
- DyLoRA model files after training are compatible with LoRA. Also, you can extract LoRA of multiple dims below the specified dim(rank) from the model file. (View Highlight)